The present invention relates generally to indexing a database, and, more particularly, to a technique for deleting duplicate records referenced in an index of a database.
It has been well known that computer systems can be used to index records of a database. In recent years, a unique distributed database has emerged in the form of the World-Wide-Web (Web). The database records of the Web are in the form of pages accessible via the Internet. Here, tens of millions of pages are accessible by anyone having a communications link to the Internet.
The pages are dispersed over millions of different computer systems all over the world. Users of the Internet constantly desire to locate specific pages containing information of interest. The pages can be expressed in any number of different character sets such as English, French, German, Spanish, Cyrillic, Kanakata, and Mandarin. In addition, the pages can include specialized components, such as embedded xe2x80x9cforms,xe2x80x9d executable programs, JAVA applets, and hypertext.
Moreover, the pages can be constructed using various formatting conventions, for example, ASCII text, Postscript files, html files, and Acrobat files. The pages can include links to multimedia information content other than text, such as audio, graphics, and moving pictures. As a complexity, the Web can be characterized as an unpredictable random update, insert, and delete database with a constantly changing morphology.
One characteristic of the World-Wide-Web makes it relatively easy to copy Web pages from one site to another. Web users frequently incorporate pages created by others into their own pages to streamline access. It is estimated that as much as 25% of the Web is composed of duplicate pages. If all the duplicate pages are fully indexed, the amount of storage required for the index would greatly increase. Therefore, there is a need for a technique which minimizes the likelihood that duplicate pages are indexed.
Briefly according to the present invention, a technique for deleting duplicate records referenced in an index of a database. In one embodiment, the technique may be realized by receiving a record; determining a fingerprint for the record; comparing the fingerprint of the record with fingerprints of previously indexed records; and, when the comparing act determines that the fingerprint of the current received record is the same as at least one of the fingerprints of any of the previously indexed records, identifying the current record as a record to be deleted.
The present invention will now be described in more detail with reference to exemplary embodiments thereof as shown in the appended drawings. While the present invention is described below with reference to preferred embodiments, it should be understood that the present invention is not limited thereto. Those of ordinary skill in the art having access to the teachings herein will recognize additional implementations, modifications, and embodiments, as well as other fields of use, which are within the scope of the present invention as disclosed and claimed herein, and with respect to which the present invention could be of significant utility.
The location entries of the first and second index entries are searched subject to one or more constraints which must be satisfied. The constraints are expressed in the general form as C(a)xe2x89xa6C(b)+K, where C(a) means a current location of the first index entry, C(b) means a current location of the second index entry, and K is a predetermined constant.
The constraints are satisfied by reading locations of the second index entry until the current location of the second index entry is at least equal to the current location of the first index entry plus the predetermined constant.